The present invention relates to an image signal processor and an image input processor.
Recently, image input processors have widely been marketed using inexpensive, light-weight single CCD chips. In such an image input processor, color filters are arranged in a mosaic pattern on a light-sensitive area to acquire color information of an object from a single image pickup element.
Such color filters are illustrated, for example, in FIG. 19 where an array of complementary color mosaic filters including cyan (Cy), magenta (Mg), yellow (Ye), and green (G). It is assumed in FIG. 19 that the luminance signals and the chrominance signals corresponding to the n-th line and the (n+1)th line of an even-numbered field are Yo,n, Yo,n+1 and Co,n, Co,n+1, and the luminance signals and the chrominance signals corresponding to the n-th line and the (n+1)th line of an odd-numbered field are Ye,n, Ye,n+1 and Ce,n, Ce,n+1. Those signals are expressed by:
Yo,n=Yo,n+1=Ye,n=Ye,n+1=2R+3G+2Bxe2x80x83xe2x80x83(1)
Co,n=Ce,n=2Rxe2x88x92Gxe2x80x83xe2x80x83(2)
Co,n+1=Ce,n+1=2Bxe2x88x92Gxe2x80x83xe2x80x83(3)
Also, Cy, Mg, and Ye are expressed using three primary colors; green (G), red (R), and blue (B), by:
Cy=G+Bxe2x80x83xe2x80x83(4)
Mg=R+Bxe2x80x83xe2x80x83(5)
Ye=R+Gxe2x80x83xe2x80x83(6)
As apparent from Equation (1), the luminance signals are generated from all the lines of both even- and odd-numbered fields. On the other hand, the chrominance signals defined by Equations (2) and (3) are generated from every alternate line while signals of dropout lines are compensated by linear interpolation. Then, proper matrix operation produces three primary color, R, G, and B, signals. This method is however disadvantageous in that the chrominance signals carry xc2xd of information as compared with the luminance signals and severe artifacts such as color moirxc3xa9 may appear on the edge. It is known for attenuating such moirxc3xa9 effects to provide a low pass filter with a crystal filter, in front of the image pickup element. The low pass filter may however decline the resolution.
As compared with simple interpolation with the chrominance signal, methods of correcting the chrominance signal with components of the luminance signal are disclosed in U.S. Pat. No. 5,032,910 and Jpn. Pat. KOKAI Publication No. 5-056446.
In U.S. Pat. No. 5,032,910, while the luminance signal Y is generated by the linear interpolation, the chrominance signal C is treated differently according to the variation of the signal Y. In the region where the luminance signal Y is less varied, the chrominance signal C is compensated by the linear interpolation, while in the region where the luminance signals is largely varied, the chrominance signal Cxe2x80x2 is restored by modifying the signal Y of that region in the following way:
Cxe2x80x2=aY+bxe2x80x83xe2x80x83(7)
where a and b are constants.
In No. 5-056446, while the luminance signal Y is generated by interpolation, the dropouts of chrominance signal Cxe2x80x2 are restored by subjecting both the luminance signal Y and the chrominance signal C to electrical lowpass filtering to have their low frequency components Ylow and Clow which are then used in Equation (8).                               C          xe2x80x2                =                  Y          ⁢                                    C              low                                      Y              low                                                          (        8        )            
The chrominance Cxe2x80x2 is equivalent to a modified luminance signal Y using the low frequency components Ylow and Clow.
However, firstly, the conventional method using the single CCD chip includes linear interpolation for the luminance signal and a combination of linear interpolation and luminance signal modification for the chrominance signal, thus causing the luminance signal to carry substantially xc2xd of information as compared with using a three CCD imaging system. Accordingly, dropouts from the chrominance signal will hardly be restored at high definition and speed.
Secondly, the conventional method restores the chrominance signal based on the luminance signal, hence requiring a more number of luminance filters than that of chrominance filters and failing to work with an arrangement of the filters at an equal ratio.
Thirdly, the conventional method includes linear interpolation for the luminance signal and a combination of interpolation and luminance signal modification for the chrominance signal. When a certain inexpensive type of optical lens system which may produce chromatic aberration and deteriorate a particular signal component is used, the method will hardly restore dropouts of the chrominance signal at high definition.
It is thus a first object of the present invention to provide an image signal processor capable of restoring dropouts of the color signal at high fidelity according to a structural model of image.
It is a second object of the present invention to provide an image signal processor capable of restoring dropouts of the color signal with high fidelity and high speed according to a structural model of image.
It is a third object of the present invention to provide an image signal processor capable of restoring dropouts of the color signal with high fidelity when the rate of the dropout of the color signal is same among different colors.
It is a fourth object of the present invention to provide an image signal processor capable of restoring dropouts of the color signal with high fidelity by repeatedly using a structural model of image.
It is a fifth object of the present invention to provide an image signal processor capable of restoring dropouts of the color signal which is generated by employing a certain inexpensive optical lens system with high fidelity by smoothing a structural model of image in accordance with the chromatic aberration.
According to a first feature for achievement of the first object of the present invention, an image signal processor for processing an image signal where each pixel is composed of more than one color signals and at least one of the color signals are dropped out according to the location of the pixel, comprises: an extracting unit for extracting from the image signal the color signals of a local area of a predetermined size which includes a desired pixel; a reference image generating unit for modifying and combining the color signals of the local area extracted by the extracting unit on the basis of a structure model and an evaluation function to generate a reference image; and a restoring unit for restoring the dropout color signal in accordance with the reference image generated by the reference image generating unit.
As a second feature for achievement of the second object of the present invention, an image signal processor for processing an image signal where each pixel is composed of more than one color signals and at least one of the color signals are dropped out according to the location of the pixel, comprises: an extracting unit for extracting from the image signal the color signals of a local area of a predetermined size which includes a desired pixel; a color signal selecting unit for calculating parameters from each of the color signals of the local area extracted by the extracting unit and selecting one of the color signals according to the parameters; a modifying unit for modifying the shape of the other color signals than the selected color signal so that their parameters are equal to the parameters of the selected color signal; a candidate image generating unit for combining the modified color signals modified by the modifying unit and the selected color signal selected by the color signal selecting unit to generate reference image candidates; a reference image selecting unit for selecting as a reference image one of the reference image candidates generated by the candidate image generating unit in accordance with a given evaluation function; and a restoring unit for restoring the dropout color signal with the use of the reference image determined by the reference image selecting unit.
As a third feature for achievement of the third object of the present invention, the image signal processor according to the first feature of the present invention further comprises an input unit having an imaging element for taking an image of an object to produce the color signals, the imaging element including an array of color filters, each color filter allocated to each pixel for the color signal, which is spatially arranged by repeatedly aligning square areas of an mxc3x97n pixel size where m and n are integers of three or higher, the square area designed so that the rate of appearance is uniform between the different color signals.
As a fourth feature for achievement of the fourth object of the present invention, the image signal processor according to the first feature of the present invention is characterized in that the restoring unit includes a luminance signal generating unit for generating a luminance signal from each of the color signal restored corresponding to the reference image, and a convergence determining unit for repeating a restoring process with the luminance signals through reviewing a profile of the luminance signals.
According to a fifth feature for achievement of the fifth object of the present invention, an image signal processor for processing an image signal where each pixel is composed of more than one color signals and at least one of the color signals are dropped out according to the location of the pixel, comprises: an input unit including an optical lens system where chromatic aberration occurs and an imaging element where the rate of appearance of the color signals is adjusted based on the amount of their chromatic aberration; an extracting unit for extracting from the image signal produced by the input unit color signals of a local area of a predetermined size including a desired pixel; an approximate reference image generating unit for modifying and combining color signals which have a low degree of the chromatic aberration in accordance with a structure model and an evaluation function to generate an approximate reference image; a smoothing unit for smoothing the approximate reference image generated by the approximate reference image generating unit in accordance with a degree of the chromatic aberration of a high aberration color signal; and a restoring unit for generating a reference image from the approximate reference image generated by the approximate reference image generating unit and the smoothed approximate reference image generated by the smoothing unit and using them to restore the dropout color signal.
Additional objects and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objects and advantages of the invention may be realized and obtained by means of the instrumentalities and combinations particularly pointed out in the appended claims.